Assessing Multisensory Integration with Additive Factors and Functional Mri

The topic of this presentation is the use of additive-factor designs in combination with functional MRI to assess multisensory integration. Unisensory and multisensory stimuli were presented across two different pairings of sensory systems, audio-visual (AV) and visuohaptic (VH). In addition to stimulus modality, signal-to-noise ratio (SNR) was included as an additive factor. Previous research investigating the effect of SNR on sensory integration has documented an effect called inverse effectiveness, where the multisensory gain increases with decreasing SNR. Potential sites of multisensory convergence were mapped for each sensory pairing and were found to be non-overlapping, suggesting that the neural mechanisms of integration are specialized for each unique pairing of sensory systems. Evidence of inverse effectiveness was found at all convergence sites, regardless of whether they were AV or VH. This result suggests that inverse effectiveness is a general characteristic of multisensory integration, regardless of the sensory pairing. The results also showed that a single-factor additive model of multisensory integration produced different outcomes at different levels of SNR. Based on this last result, we conclude that an additive-factors approach to assessing multisensory integration will provide more reliable inferences than single-factor designs. Advances in neuroimaging in the last two decades have provided methods for studying the neural mechanisms of cognitive function in humans non-invasively. Functional MRI and other neuroimaging techniques have provided unique insights about the relations between cognitive function and brain circuits. The relations between fMRI measurements, such as blood oxygenation-level dependent (BOLD) response, and neural activity measurements, such as action potentials or local field potentials, however, are not fully understood. Thus, caution must be exercised when making inferences based on neuroimaging measurements. The study of multisensory perception has a long history in science (Molyneux, 1688), but recently there has been increased interest in multisensory phenomena, and especially their neural mechanisms (Amedi, Von Kriegstein, Van Atteveldt, Beauchamp, & Naumer, 2005; Stein & Stanford, 2008). Our research program focuses specifically on multisensory object perception, including the perception of object identity through vision, touch, and hearing. The focus of this presentation is twofold. First, to examine whether or not there are unique neural substrates involved in integrating information from unique pairings of sensory systems or, more specifically, are the sites of integration for visuo-haptic pairings different from the sites for audio-visual pairings. The second focus is to evaluate the utility of using additive-factors designs to assess multisensory integration with BOLD fMRI measurements. The additive factor used was stimulus signal-to-noise ratio (SNR). This factor was chosen because it is commonly used in multisensory research to study a phenomenon called ‘inverse effectiveness’ (Meredith & Stein, 1986). In both single-unit recordings and behavioral measures (accuracy and RT), inverse effectiveness is observed when multisensory gain increases as stimulus SNR decreases (Holmes, 2007). In other words, as the stimuli become more difficult to discriminate, there is a greater benefit to integrating across sensory sources.